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The optimal filtering of Markov jump 
processes in additive white noise. 



This note is based on Wonham [T] . The differences between this note and [Tj are discussed 
in Section VIII. 



I. Statement of the Problem . 

Let x{t) be Markov jump process with stationary transition probabilities and with a finite 
number of states. Let ai,a2, ■ ■ ■ ,aK be the states, let Pij{h) be the transition probabilities 



Pij{h) = Prob + h) = aj x{t) = aj| . 



Let 



Pij{h) = < 



l — Vih + o{h) i=j, /i\,0 
, i>ij{h)+o{h) i^j, /i\,0 



K 



where Vi > 0, Vij > and Ui = Uij ,i = 1, . . . , K. 



Let Pi{0) be the initial distribution of x{0). In addition let y{t) be a process given by 
dy{t) = x{t) dt + (5 dw{t) , y(0) = 
where (3 is known and w{t) is a standard Brownian motion. The problem is to find 

Pj{t) =Vmh\x{t) = aj 7r^y(-)} 



where vr^y(-) stands for a<s<b. 
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II. A general expression for Pj{t) 

The conditional probability Prob = c 

same as the conditional expectation of the function 6ij {j fixed) given A. Applying Doob's 
theorem 8.8 (|2i p. 21]), there exists a sequence ti,t2, ■ ■ ., all in [0,t], such that a.s. 



A 



I (where 



A is some condition) is the 



Pj{t) = FToh[x{t) = aj I y{h),y{t2),...] 



and by the martingale convergence theorem (|2l Corl p. 332]) a.s. 



Pj{t) = lim ef'\t) 

n— >oo 

where ef'\t) = Prob |x(t) = 
In the following we will use: 



(1) 



y(*i),y(i2), • • • ,y(in)} • 



Consider now a fixed t^, < < t; let | y(t^)) be the probability that x{t) = aj given 
y{t^) and let be 

C/. = / x(t) (it . 



Then 



^ fOO 

J2Pi(^)Pvii) / Piiyitf^)-^i^)P2{Ci.\xiO) = ai, x{t) = aj)dC, 
i=i -^-"^ 



K 



^^(The same expression as in the numerator) 



Note that Pi is normal (0,/3^t^). Let x{t) be a process independent of x{t) and w{t) with 
the same law as x{t). Let ^(s) = J^x{t) dt, then 
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K 



Pj 



(t I yit 



i=l 



{yit,)-Cit,)r 

2P% 



x{0) = ai, x{t) = aj 



I K K . 
j=i i=i [ 



2P% 



x{0) = ai, x{t) = aj 



The conditioning of the expectations in the above expression are all x{s) paths which start 

at s = with Oj and terminate at s = i in the state aj. Now let Sr^n = fji-, r = 0, 1, . . . ,n. 
Let 

r}r,n = y{Sr+l,n) " y{Sr,n) 



ds 



Then by the same arguments as above 
Pj (*|y(''n)' r = 0,l,...,nj = 



K 



n— 1 



{jlr,n ^r,nY 



^p.(0)p.,(t)E|exp-X;^:^^ 



x(0) =aj, x{t) = aj 



K 



(2) 



(numerator) 



The argument of the exponential is 

ra-l 



r=0 



The first term will be cancelled by the same term in the denominator. The last term 
converges a.s. as n ^ oo to 

ft 



1 

W'' Jo 



,2 , x{tfdt 



The middle term converges a.s. to 

-1^ J^x{t)dy{t). 



We want to apply these results to the evaluation of the limit of the numerator of ^ as 
n — > oo. In order to do that we have to show that if /„ — > / as n ^ cxd, then E{fn \ ) — > E{f \ ) . 
Since 



exp 



n-l 



r,n 

< exp [>l^^^-t + 2A^ax-sup{|y(ti)-y(t2)|, h,t2£[0,t]}] 
it follows, by dominated convergence that the limit of the numerator of ([2]), as n — > oo, is 



V'j(i) = ^Pi(0)pij(t)-E|exp j^x'^{s)ds + ^ j^x{s)dy{s) 



x(0) = aj, x{t) = aj 

(3) 

and the conditioning is with respect to all the paths which start at x{0) = ai and terminate 
at x{t) =aj. Similarly the limit of the denominator is Ylf=i'^j{'^) where il^j{t) is given by 
equation Since ■ipj{t)>0 a.s. we have 



lim pj{t I y(r^), r = 0,l,... 



nj 



(4) 



The limits ([3]) and (jH) were obtained by a particular sequence of partitions of [0,t], but 

it is clear that the same result will hold for any sequence of partitions {sr,n} such that 

= so,n < si,n < ■ ■ ■ < Sn.n = t and such that max (sr+i,n — Sr,n) ^ as n ^ oo. We may 

r 

therefore use a sequence for which ([1]) is true. Therefore 



(5) 



4 



III. The stochastic differential equation for ^j{t). 







jexp 





1 f \^ 1 /" ^ 

x'^{s)ds + ^J x{s)dy{s) 



Let tpij{a,b), {b>a) be 
ipij{a, b) = pij{b-a) E |exp 
Then, comparing with ([3]): 

i=l 

Consider a fixed reahzation of ■K^^y{-), we prove now that 

K 

i^j{t+h) = Y,Mt) ■ 4^i,j{t,t+h) . 



K 



1=1 



Proof: 

V'ij(o,t+/i) 





i-t+h i-t+h ■ 




jexp 


-/ ... +/ ... 






.JO JO 





x{a) = ai, x{b) =aj 



(6) 



K 





I-t+h I-t+h ■ 




jexp 


- ... +/ ... 






.JO JO . 





x{t + h) = aj, x{0)=ai 

x{t+h)=aj, x{t) = ak, x(0) = aij> 
Prob|x(t) = afe | x{t + h) = aj, x{0)=ai^ . 



Since x{t) is a Markov process, the conditional expectation becomes the product of two 
conditional expectations (since, given x{t), x{t—a) and x{t+P) are independent for a, (3>0). 
Moreover 

Pik{t)pkj{h) 



Prob 



= ttk I x{t+h) = aj, x(0)=aj| 



Pij{t+h) 
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Therefore 



2Pij{0,t + h) 



y2pik{t)ELxp [-/... +/ 

k=i I L ^0 ^0 









|exp 


[-/' 






. ^0 


Jo 



x{t) = ak, x{0) = a,, 
x{t+h) = aj, x{t) = ak 



K 



k=l 

which is the required result. 

Since the x{t) process is Markov and the w{t) process has independent increments, it follows 
that ipi^kiO, t) and ipkjit, t-\4i) are conditionally independent given x{t). Therefore the process 
{x{t),ilji{t),ip2{t), ■ ■ ■ ,ijjK{t)) is a K+1 dimensional Markov process. 

ijjij{t,t+h) will now be evaluated for small h. Assuming that h is small enough so that 
the possibility that more than one transition in [t,t+h] can be ignored we have 

'4)i^i{t,t+h) = (l-fi/i)exp 
tpij(t,t+h) = Vij h exp 



-^h + ^iyit+h)-yit)) 



The factors 6i and O2 were included in the last expression in order to indicate that it is 
unknown where in \t,t+h\ the transition occurred; it will turn out that this is immaterial. 

Setting now y{t+h)—y{t) = Jl^^x{s) ds + (3 w{t+h) — (3 w{t) and expanding the exponential 
in a power series we obtain 

i^,,i{t, t+h) ^l-u,h-^h + ^ (y(t+h)-y{t)) + ^ {w(t+h)-w{t)f + o,ih, {Ahwf) 
where oi{h, (Ahw)^) denotes the terms omitted. Also 
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ijij{t,t+h) ^ Uij h + 02{h, (Ahwf) . 
Substituting into ([6]) we have 

^l;J{t + h) = ^j{t)-vji:j{t)h--^h^l,j{t) + ^^^!^{y{t + h)-y{t)) 



a? 



K 



+ Tj4 {w{t + h)-w{t)f + iPiit) V,j h + 03(/l, {^hwf) 



1=1 



Therefore 

V'iW-V'i(o) 



Jo 



K 



1=1 



2/32 



+ 



t/h 



' "2 ( w((r + l)/i)-w(r/i) )^ 



*"i^dy(.) + hm y-M^'-' — ^2 



.0 ^ 2/32 

r=l 



+ hm Vo3(/i,(A;,u;)2) . 
h— >o — ' 



(7) 



r=l 



The first sum can be shown [6] to converge a.s. to 



The second sum can be shown [3j to converge to 0. (For example, if |/(s)| <M in [0, t] then 
the sum Yll=if{^f^) ^ {w{{r+l)h)~w{rh)) is bounded by 

Mt sup 

t 

but since is a.s. continuous it is also uniformly continuous and the last term converges 
to zero as /i— >0, hence the sum converges a.s. to zero). 



Equation ([7]) becomes: 

dijjj (t) = —i^j (t) dt + y^ Vij ^i{t) dt + 



K 



i=l 



(8) 
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with the initial condition ipj{0)=pj{0). 



K 

Let <i){t) = ^Vj(i)- Then 



K 



i=i ^ 



and 



Mt) = = (9) 

Equation ([8]) is a stochastic differential equation for ^j(t) from which Pj{t) can be obtained 
by (©. 

The Langevin equatiorE corresponding to ([8]) can be derived using equation (4.30) of [5]; 
the result is 

and dy{t)/dt is x{t) plus "white noise". 

IV. The stochastic differential equation for pj it) . 

Since (by definition of V'j and (j)) (p{t)^0 a.s., we may apply Ito's rule of differentiation [3] 
to dSD: 

dp-(t) = '<Pj{t)dHt) 1 ajipjjt) ( aiipi{t) \ ^2^^ 



K - 2 



m P' I 

Substituting for d'4)j and dcj), and setting ^^"^"f)'^''^*^ = ^(^) we get 



^see [3] or I2] 



K 

dPj {t) = -Vj pj (t) dt + '^pi (t) Uij , dt 



+ p-^{aj-x{t))pj{t)dy{t) (11) 
+ /3-^xit)pj{t) {aj-xit)) dt [x{t)^j:f=iajPjit)] 



which are the equations derived by Wonham. Note that as — > oo we get the Kolmogorov 
forward equation (as expected). The equations for ipj{t) are more elegant than those for 
Pj{t). However, the equations for Pj{t) are probably more useful for applications since they 
ensure that the output is always in [0, 1] (while tpj{t) can be anywhere in (0, oo))- It seems 
also that perhaps the Pj{t) may have a stationary distribution while il>j{t) may not have 
such a distribution. 

The Langevin equation corresponding to ([TT]) is, by eq. (4.30) of [5j, 
dpj{t) _ 



i^j \ 1=1 / 



dt 

+ p-\a,-x)p,^. (12) 

V. Example - The random telegraph signal. 

In this case ai = 1, 02 = —1 

Vi = Vij = v\ i,j = 1,2 
where v is the expected number of jumps. 
Let qit) = px(t) - p2(t) 

then x(t) = q{t) and the equations of the last paragraph become 
dq{t) = -2uq{t) dt - P'^q{t){l - q\t)) dt + /3-^{l-q\t)) dy{t) 
or, equivalently 

dq{t) = -2vq{t)dt- l3-'^q{t){l-q\t))dt + f3-'^{l-q\t))x{t)dt 
+ f3-\l-q\t))dw{t) . 
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The Langevin equivalent of this stochastic differential equation is given by ( [5] , [7] ) 

dq{t) 



dt 



-2u qit) + p-'{l-q\t)) x{t) + r\l-q\t)) n{t) 



where n{t) is "white noise". Let x(t)+n(t) = r(t), then we have the Riccati equation: 
dq{t) 



dt 



-2vq{t)+(3-\l-q\t))r{t) 



The physical filter to compute q{t) will therefore be 



[r{t) = x{t) + n{t) 



initial condition g(0) 



multiplier 





2 A amplifier 




I I 

gW.i ... I 

— — decision r 



or: 



Tit) 




linear network with 
transfer function 
1 




I I 

gW.i ... I 

— decision r 
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If, instead of the analog filter we use a digital computer we have to distinguish between 
two cases. Let w„ be the cutoff frequency of the "white noise" and uJs be the sampling 
frequency of the computer. Case 1: i^<CcJn^'^s! Case 2: z^<CWs<Ca;„. It follows from [5] 
and [7J that in case 1 the computer should be programmed to solve the Langevin equation. 
In case 2, Maruyama's approximation theorem is applicable [5j and the computer should be 
programmed to solve Ito's equation (via Maruyama's approximation). 

An error analysis for this example is discussed in [Ij. 

VI. Some transformations on tpjit). 
Equation (flUj) can be rewritten as 



dt 



ai 



a2 











UK 



f3^ 



■m 



(lOa) 



where ^ is a constant matrix and ip is the vector [ipi, . . . , i^k)'^. 
Setting 

T{t) = e-^* • m 
we get the Langevin equation for T(t): 



dm ^ 

dt 



a2 











UK 



e^* r(t) 



(13) 



Setting 9j{t) = log^pj{t) 
hence 



dOjit) 
dt 



1 



(14) 



and since 0i does not appear in front of the last term, this is the Ito as well as the Langevin 
equation for 9j{t) . 
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VII. The prediction of x{t). 

The problem is now to find the probability that x{t+h) = aj, h>0, conditioned on ttqU^-) 
The result follows directly from 



Prob <^ x{t+h)eA 



^Prob<^ 



■-Ui 



Troy{-)€B\ ■ Prob<^ x(t + /i)GA 



x{t) = ai, ■KQy{-)eB 



Since Prob <x{t+h) 



x{t), 7rly{-) \ = Fioh \x{t+h) 



> , it follows that (as expected) 



Prob < x{t + h) = aj 



K 



(15) 



1=1 



where Pi{t) is the solution to (jlip and Pij{h) are defined at the beginning of section I. The 
extension of ()15p to the probability distribution of functionals on x(s), s>t, conditioned 
on ir^yi') is obvious. 



VIII. Remarks . 

Section I, II and the first halves of III and V follow from Wonham [1]. Instead of proceed- 
ing directly to obtain the stochastic differential equations for Pj{t) as done in [1] we first 
derive the stochastic differential equations for tpjit) (section III) from which the stochastic 
differential equations for Pj{t) are derived by a singular transformation (section IV). The 
equations for ipj{t) are considerably simpler and are of a standard form (section VI). It is 
believed that a similar approach can be used in the case treated by Stratonovich and Kush- 
ner (where x(t) is a diffusion process). The treatment in this note is restricted to /? = const, 
the treatment in |lj is for (5 = (5{t) where (5{t) is continuously differentiable and bounded 
away from zero. The extension of the arguments and results of this note to (3 = f3{t) is 
straightforward. 
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A question which was left open in ^ was the problem of the realization of the results 
as physical "filters". Recent work reated to this problem [5], [7j gives answers to this 
question. Equation (4.30) of [5], which was used in this note, was derived in |5j by a 
heuristic argument. Unpublished calculations (for piecewise linear approximations to the 
Brownian motion) show that (4.30) is correct. A short discussion on the realization problem 
is included in section V. 
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